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DETAILED ACTION 

I. Response to Arguments 

Applicants' response to the last Office Action, mailed 17 May 2007, has 
been entered and made of record. 

The rejections of claims 4 and 29-31 are rendered moot by applicant's 
cancellation of those claims. 

Applicant's arguments, see pages 13-15 of Applicant's remarks, filed 27 
August 2007, with respect to the rejection(s) of claims 1-3, 5-28, and 32- 
37 under 35 U.S.C. §103, in view of Katayama, have been fully 
considered and are persuasive. Examiner is persuaded by Applicant's 
assertion that there would not have been a reasonable expectation of 
success by combining the three-dimensional object extraction method 
suggested by Katayama with the three-dimensional panoramic mosaic 
methods of Szeliski. Therefore, the rejection has been withdrawn. 
However, upon further consideration, a new ground(s) of rejection is 
made in view of the combination of Szeliski, Luken, Seago, and Blank, as 
presented in the office action below. 
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II. Claim Rejections - 35 USC § 1 12 

A. Second Paragraph - Indefinite 

The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

Claim 37 is rejected under 35 U.S.C. 112, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the 
subject matter which applicant regards as the invention. 

Claim 37 recites, "wherein the one or more editing tools comprises a 
ground plane tool, an extrusion tool, a depth chisel tool, and a non- 
uniform rational B-spline tool." It is unclear how one editing tool is 
comprised of multiple editing tools. Examiner believes Applicant intended 
for this limitation to read in the alternative, similar to claim 18, and will 
examine the claim as such. 

B. Sixth Paragraph - Means-Plus-Function Interpretation 

Regarding independent claim 32, the means-plus-function language will 
be interpreted as follows: 

• Means for receiving one or more image panoramas representing a 
visual scene having one or more objects relates to a device "such as a 
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scanner, a digital camera, or other means for receiving, storing, 
and/ or transferring digital images such one or more image 
panoramas, two-dimensional images, and three-dimensional images," 
as defined in paragraph 1 12 on page 27 of the specification. 

• Means for allowing a user to interact with the system relates to a 
mouse or other pointing and dragging device such as a track ball to 
orient the panorama, as defined in paragraph 84 on page 18 of the 
specification. This interaction means is interpreted to be the means 
for determining a directional vector for each image panorama and 
aligning the image panoramas relative to each other. 

• Means for creating a three-dimensional model from the aligned 
panoramas relates to interactive software tools, for defining the 
geometries and textures of elements within the image as defined in 
paragraph 99 on page 23 of the specification. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 

all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or 
described as set forth in section 102 of this title, if the differences between the subject matter 
sought to be patented and the prior art are such that the subject matter as a whole would 
have been obvious at the time the invention was made to a person having ordinary skill in the 
art to which said subject matter pertains. Patentability shall not be negatived by the manner 
in which the invention was made. 
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A. Claims 1-3, 5-6, 8-10, and 32-35 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski et al. (US Patent Number 6,157,747, herein referred to 
as Szeliski), in view of Luken (US Patent Number 5,923,334), in further view of 
Seago (US Patent Number 5,990,900). 

Regarding independent claim 1, a computerized method for creating a 
three dimensional model from one or more image panoramas, the method 
comprising: 

• Receiving one or more image panoramas representing a visual scene 
and having one or more objects (See Szeliski: Fig. 6; Col. 4, Lns. 66- 
67, wherein the invention creates full view panoramic mosaics from 
image sequences. Col. 11, Lns. 16-21 teaches the presence of objects 
within the panoramic images.); 

• Determining a directional vector indicating an orientation of the visual 
scene with respect to a reference coordinate system (See Szeliski: Fig. 
15; Col. 22, Lns. 4-9, wherein the unit vector indicates the ray 
direction in the final composite image mosaic); 

• Transforming the image panoramas such that the directional vectors 
are substantially aligned relative to the reference coordinate system 
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(See Szeliski: Figs. 11 and 15; Col. 22, Lns. 2-4, wherein a bundle of 
rays is adjusted so they converge to one point.); 

• Aligning the transformed image panoramas to each other (See 
Szeliski: Fig. 17; Col. 23, Lns. 10-12, wherein minimizing the 
differences between ray directions aligns the images.); and 

• Creating a three dimensional model of the visual scene from the 
. transformed and aligned image panoramas using the reference 

coordinate system (See Szeliski: Col. 5, Lns. 15-19, wherein the three- 
dimensional model of the scene is the texture-mapped polyhedron 
surrounding the origin, which can be viewed or explored using 
standard 3D graphics viewers.). 

Szeliski does not expressly disclose determining a directional vector for 
each image panorama. Nevertheless, Luken is cited for teaching eight 
direction vectors D0-D7 associated with six rectangular images mapped 
to the inside of an octahedron (See Luken: Figs. 7-10, 14 and 17; Col. 7, 
Lns. 28-36, wherein it is determined which of the six rectangular images 
is intersected by one of the eight direction vectors.). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used direction vectors for each image within an 
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environment map that is mapped to the sides of an octahedron, as 
taught by Luken, with the three-dimensional model environment map of 
the visual scene, as taught by Szeliski, because Luken: (1) is directed to 
the same problem of using polyhedral environment maps to create and 
view three dimensional images from data representing multiple views of a 
scene; (2) is in the same field of endeavor of image processing systems; 
and (3) Luken expressly suggests that the direction vectors provide an 
efficient system for generating and viewing three-dimensional panoramic 
images based environment maps, and offer an improved level of 
interactive graphical feedback (See Luken: Col. 3, Lns. 5-8). 

Though Szeliski teaches creating a three-dimensional model of the visual 
scene (See Szeliski: CoL 5, Lns. 15-19, wherein the three-dimensional 
model of the scene is the texture-mapped polyhedron surrounding the 
origin.), Szeliski does not expressly suggest creating three-dimensional 
objects within the scene by individually modeling selected objects within 
the scene, and thus does not expressly suggest that creating a three 
dimensional model includes identifying a selected object in the 
transformed and aligned image panoramas and associating geometry 
information with the selected object, the geometry information 
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comprising 3-D coordinates describing the position and orientation of the 
selected object in the reference coordinate system. 

Seago is cited for converting digitized two-dimensional images of objects 
into three-dimensional digital computer objects (See Seago: Col. 2, Lns. 2- 
5). Specifically, with reference to figure 2, Seago begins the process with 
a digitized two-dimensional perspective image of one or more objects (See 
Seago: Fig. 2, Item 40; Col. 4, Lns. 34-36). Next, an object contained 
within the digital image is selected for conversion into a three- 
dimensional object by a user (See Seago: Fig. 2, Item 42; Col. 4, Lns. 46- 
53). The orientation of a three-dimensional coordinate space for the 
selected object is derived based on the calculated vanishing points, 
wherein the three-dimensional coordinate space is the screen world 
coordinate space (SWCS) with the origin located at the center of the 
perspective image (See Seago: Fig. 2, Item 50; Col. 5, Lns. 24-38). 
Therefore, Seago sufficiently teaches the claim limitations of creating a 
three dimensional model by identifying a selected object in a digital, two- 
dimensional image and associating geometry information with the 
selected object, the geometry information comprising 3-D coordinates 
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describing the position and orientation of the selected object in the 
reference coordinate system. 

It is noted that, while the exemplary embodiment of Seago is an 
implementation of the invention using only a single input image, the 
system can also use two or more input images. In this case the steps 
illustrated in figures 6A and 6B are applied to all images (See Seago: Col. 
11, Lns. 45-47). Seago merely requires an input of perspective images in 
order to properly implement the disclosed method. Szeliski teaches that, 
"any deviations from the pure parallax-free motion model or ideal pinhole 
(perspective) camera model may result in local misregistrations, which 
are visible as a loss of detail or multiple images (ghosting) (See Szeliski: 
Col. 3, Lns. 36-39)." This clearly conveys the intention of Szeliski to use 
perspective images as the input to the panoramic system. Since the 
images used in the image panoramic mosaics are perspective images, the 
methods taught by Seago to extract three-dimensional objects from 
perspective images could be implemented with the teachings of Szeliski 
to extract three-dimensional images from a panoramic mosaic of 
perspective images. 
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Therefore, it would have been obvious to one of ordinary skill in the art, 
at the time the invention was made, to have used the three-dimensional 
object extraction method, as taught by Seago, to extract three- 
dimensional objects from the perspective images of the panoramic 
mosaic, as taught by Szeliski, as modified by Luken, because Seago is in 
the same field of endeavor of generating three-dimensional computer 
graphics using image capture devices; and Seago expressly suggests that 
this method of object extraction produces accurate three-dimensional 
objects more efficiently than conventional systems that try to extract the 
three-dimensional objects using analytical mathematical interpretations 
and orthogonal image analysis (See Seago: Col. 11, Lns. 49-55). 

Regarding independent claim 32, Szeliski teaches a system for creating a 
three dimensional model from one or more image panoramas, the system 
comprising: 

• Means for receiving one or more image panoramas representing a 
visual scene having one or more objects (See Szeliski: Fig. 2B, Item 
210; Col. 9, Lns. 6-9, wherein the means for receiving image 
panoramas is a digital /electronic camera, a still camera, or a video 
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camera,, or film or photographic scanner, which captures a sequence 
of images.); 

• Means for allowing a user to interact with the system to determine a 
directional vector for the image panorama (See Szeliski: Fig. 2A, Items 
40 and 42; Col. 8, Lns. 30-34, wherein a user may enter commands 
and information into the personal computer 20 through input devices 
such as a keyboard 40 and pointing device 42. Other input devices 
(not shown) may include a microphone, joystick, game pad, satellite 
dish, scanner, or the like.); 

• Means for aligning the image panoramas relative to each other (See 
Szeliski: Fig. 2 A, Items 40 and 42; Col. 8, Lns. 30-34, wherein a user 
may enter commands and information into the personal computer 20 
through input devices such as a keyboard 40 and pointing device 42. 
Other input devices (not shown) may include a microphone, joystick, 
game pad, satellite dish, scanner, or the like.); and 

• Means for creating a three-dimensional model from the aligned 
panoramas (See Szeliski: Col. 27, Lns. 64-66, wherein the shape of 
the model and the embedding of each face into texture space are left 
up to the user.). 
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Szeliski does not expressly disclose determining a directional vector for 
each image panorama. Nevertheless, Luken is cited for teaching eight 
direction vectors D0-D7 associated with six rectangular images mapped 
to the inside of an octahedron (See Luken: Figs. 7-10, 14 and 17; Col. 7, 
Lns. 28-36, wherein it is determined which of the six rectangular images 
is intersected by one of the eight direction vectors.). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used direction vectors for each image within an 
environment map that is mapped to the sides of an octahedron, as 
taught by Luken, with the three-dimensional model environment map of 
the visual scene, as taught by Szeliski, because Luken: (1) is directed to 
the same problem of using polyhedral environment maps to create and 
view three dimensional images from data representing multiple views of a 
scene; (2) is in the same field of endeavor of image processing systems; 
and (3) Luken expressly suggests that the direction vectors provide an 
efficient system for generating and viewing three-dimensional panoramic 
images based environment maps, and offer an improved level of 
interactive graphical feedback (See Luken: Col. 3, Lns. 5-8). 
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Szeliski does not expressly disclose creating a three dimensional model 
by identifying a selected object in the aligned image panoramas and 
associating geometry information with the selected object, the geometry 
information comprising 3-D coordinates describing the position and 
orientation of the selected object in a reference coordinate system. 

Seago is cited for converting objects within digitized two-dimensional 
images into three-dimensional digital computer objects (See Seago: Col. 
2, Lns.2-5). Specifically, with reference to figure 2, Seago begins the 
process with a digitized two-dimensional perspective image of one or 
more objects (See Seago: Fig. 2, Item 40; Col. 4, Lns. 34-36). Next, an 
object contained within the digital image is selected for conversion into a 
three-dimensional object by a user (See Seago: Fig. 2, Item 42; Col. 4, 
Lns. 46-53). The orientation of a three-dimensional coordinate space for 
the selected object is derived based on the calculated vanishing points, 
wherein the three-dimensional coordinate space is the screen world 
coordinate space (SWCS) with the origin located at the center of the 
perspective image (See Seago: Fig. 2, Item 50; Col. 5, Lns. 24-38). 
Therefore, Seago sufficiently teaches the claim limitations of creating a 
three dimensional model by identifying a selected object in a digital, two- 
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dimensional image and associating geometry information with the 
selected object, the geometry information comprising 3-D coordinates 
describing the position and orientation of the selected object in the 
reference coordinate system. 

Therefore, it would have been obvious to one of ordinary skill in the art, 
at the time the invention was made, to have used the three-dimensional 
object extraction method, as taught by Seago, to extract three- 
dimensional objects from the perspective images of the panoramic 
mosaic, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). 

Regarding claim 2, as it depends from claim 1, Szeliski teaches the 
directional vector is determined based, at least in part, on instructions 
identifying elements of the image panorama received from a user (See 
Szeliski: Col. 8, Lns. 30-32 and Col. 27, Lns. 64-66). 
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Regarding claim 3, as it depends from claim 2, Szeliski teaches a method 
wherein the instructions from the user identify two or more substantially 
parallel features in the image (See Szeliski: Col. 20, Line 64 - Col. 21, 
Line 6 and Lines 25-34). 

Regarding claim 5, as it depends from claim 2, Szeliski teaches a method 
wherein the instructions from the user identifying a horizon line of the 
image panorama (See Szeliski: Fig 4: Col. 9, Lns. 54-62). 

Regarding claim 6, as it depends from claim 2, Szeliski teaches a method 
wherein the instructions comprise the identification of two or more areas 
of the image, each area containing one or more elements and further 
comprising automatically identifying the two elements contained in the 
two or more areas (See Szeliski: Fig. 6; Col. 20, Line 49 - Col. 21, Line 
24, wherein a feature-based point correspondence is established between 
a pair of images by dividing each image into patches and identifying 
prospective "feature" points within the patches.). 

Regarding claim 8, as it depends from claim 1, Szeliski does not 
expressly suggest that the image panoramas are aligned relative to the 
reference coordinate system such that the directional vector is at least 
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substantially parallel to one axis of the reference coordinate system. 
Nevertheless, Luken is cited for determining for each of the eight 
direction vectors DO, Dl ,D2 ... D7, which of the six axis-aligned, 
rectangular images is intersected by the direction vector Di (See Luken: 
Fig. 7, Item 707; Col. 6, Line 40— Col. 7, Line 36). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have aligned the direction vector with an axis of the reference 
coordinate system, as taught by Luken, with the panoramic mosaic 
method, as taught by Szeliski, because the numerical complexity of a 
vector representation is reduced if the vector is aligned with an axis. 

Regarding claim 9, as it depends from claim 1, Szeliski does not 
expressly suggest that the image panoramas are aligned relative to the 
reference coordinate system such that the directional vector is at least 
substantially orthogonal to one axis of the reference coordinate system. 
Nevertheless, Luken is cited for determining for each of the eight 
direction vectors DO, Dl ,D2 ... D7, which of the six axis-aligned, 
rectangular images is intersected by the direction vector Di (See Luken: 
Fig. 6A and 7, Item 707; Col. 6, Line 40— Col. 7, Line 36). Since the six 
rectangular images are axis aligned, then a directional vector that is 
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parallel to one axis must be perpendicular to the other two spatial axes. 
In other words, in order for a directional vector to intersect one of the 
rectangular images, that vector must be substantially parallel to one 
axis, which requires it to be substantially perpendicular to the others. It 
would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have aligned the direction vector with an axis 
of the reference coordinate system, as taught by Luken, with the 
panoramic mosaic method, as taught by Szeliski, because the numerical 
complexity of a vector representation is reduced if the vector is 
perpendicular to an axis because it suggests that the vector is parallel 
with another axis. 

Regarding claim 10, as it depends from claim 1, Szeliski teaches a 
method wherein the image panoramas are aligned according to 
instructions received from a user (See Szeliski: Col. 27, Lns. 64-66, 
wherein the user aligns the image panoramas into texture space.). 

Regarding claim 33, as it depends from claim 32, Szeliski teaches a 
system wherein the input images comprise two-dimensional images (See 
Szeliski: Figs. 3-4; Col. 9, Lns. 56-58, wherein a camera 310 having its 
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optical center fixed at point C (FIG. 3) captures a sequence of 2D still 
images Io, h, I2, I3 .). 

Regarding claim 34, as it depends from claim 32, Szeliski teaches the 
input images comprise three-dimensional images including geometry 
information (See Szeliski: Figs. 3-4 and 6; Col. 9, Lns. 56-62, wherein the 
camera captures a sequence of 2D still images (Io, Ii, h, I3.) as it pans, 
the center rays of these images being focused on 3D points (Po, Pi, P2, 
P3...) at a focal length f from the optical center point C. The points Pi are 
defined relative to a fixed 3D world coordinate system (P x , P y , P z ). Since 
the three-dimensional images correspond to the two-dimensional images 
that include depth information in the form of focal length. The geometry 
information is the inverted V shape shown in both figures 4 and 6.). 

Regarding claim 35, as it depends from claim 32, Szeliski teaches 
aligning the image panoramas according to instructions received from a 
user (See Szeliski: Col. 27, Lns. 64-66). 

B. Claims 11-12, 22-28, and 36 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski in view of Seago. 
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Regarding independent claim 1 1 , Szeliski teaches a computerized method 
of interactively editing objects in a panoramic image, the method 
comprising: 

• Receiving an image panorama representing a visual scene, the image 
panorama having one or more objects (See Szeliski: Fig. 6; Col. 4, Lns. 
66-67, wherein the invention creates full view panoramic mosaics 
from image sequences. Col. 11, Lns. 16-21 teaches the presence of 
objects within the panoramic images.) and a point source (See 
Szeliski: Fig. 3, Item 310; Col. 9, Lns. 56-57, wherein the point source 
is camera 310 having its optical center fixed at point C); 

• Creating a three dimensional model of the visual scene using features 
of the visual scene and the point source (See Szeliski: Col. 5, Lns. 15- 
19, wherein the three-dimensional model of the scene is the texture- 
mapped polyhedron surrounding the origin (camera location or point 
source), which can be viewed or explored using standard 3D graphics 
viewers.); 

Szeliski does not expressly disclose: 

• Identifying a selected object in the image panorama and associating 
geometry information with the selected object, the geometry 



Application / Control 
Number: 10/780,500 
Art Unit: 2628 



Page 20 



information comprising 3-D coordinates describing the position and 
orientation of the selected object in a reference coordinate system; 

• Receiving an edit to the selected object in the panorama; 

• Transforming the edit relative to a viewpoint defined by the point 
source; and 

• Projecting the transformed edit onto the selected object. 

Seago is cited for converting digitized two-dimensional images of objects 
into three-dimensional digital computer objects (See Seago: Col. 2, Lns. 2- 
5). Specifically, with reference to figure 2, Seago begins the process with 
a digitized two-dimensional perspective image of one or more objects (See 
Seago: Fig. 2, Item 40; Col. 4, Lns. 34-36). Next, an object contained 
within the digital image is selected for conversion into a three- 
dimensional object by a user (See Seago: Fig. 2, Item 42; Col. 4, Lns. 46- 
53). The shapes or polygons that define the selected object's sides relates 
to the geometry information, and are determined based on user 
designated vertices or vanishing lines at significant features of the 
selected object (See Seago: Fig. 2, Item 50; Col. 5, Lns. 35-38). Once all 
the polygons and plane indexes of a selected object have been 
determined, a three-dimensional object oriented within the selected 
object's three-dimensional coordinate space is determined (See Seago: 
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Fig. 2, Item 54; Col. 5, Lns. 50-53). Therefore, Seago sufficiently teaches 
the claim limitation of creating a three dimensional model by identifying 
a selected object in a digital, two-dimensional image and associating 
geometry information with the selected object, the geometry information 
comprising 3-D coordinates describing the position and orientation of the 
selected object in the reference coordinate system. 



Furthermore, Seago teaches receiving an edit to the selected object in the 

panorama, transforming the edit relative to a viewpoint defined by the 

point source, and projecting the transformed edit onto the selected object 

in figure 3 and column 6, lines 1-34, as repeated in the following excerpt: 

"The viewpoint is along a normal to the image plane at the image's center. The 
three-dimensional coordinate space (SWCS) is set at the center of the perspective 
image. Next at block 62, the intersection points between the image plane and the 
line-of-sight from the viewpoint to the endpoints of line segments extended to an 
approximate infinite point along each axis of the three-dimensional coordinate 
space (SWCS) are determined. In other words, the end of each axis is projected back 
to the image plane. At block 64, each calculated vanishing point is compared to the 
corresponding determined projected axis endpoint." 

"However, if the non-linear optimization algorithm determines that the error function is 
not at an acceptably low level, the three-dimensional coordinate space orientation 
and viewpoint focal length are adjusted based on the determined error function." 

The edit instruction is received when decision block 70 in figure 3 
branches along the "NO" path, indicating that the result of the error 
function is not sufficiently low. The edit is executed by adjusting the 
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orientation of the three-dimensional coordinate space. Once the edit is 
complete, the process loops back to block 62 in which the results of the 
edit in three-dimensions are transformed into 2D coordinates and 
projected back to the image plane along the line-of-sight. The line-of- 
sight corresponds to the viewpoint, and therefore the transform of the 
edit occurs relative to the viewpoint defined by the point source. 

As previously noted, the exemplary embodiment of Seago only uses a 
single input image as input for the extraction process. Nevertheless, 
Seago teaches that the system can also use two or more input images. In 
this case the steps illustrated in figures 6A and 6B are applied to all 
images (See Seago: Col. 11,. Lns. 45-47). Seago merely requires the input 
images to be perspective images in order to properly implement the 
disclosed method. Szeliski teaches that, "any deviations from the pure 
parallax-free motion model or ideal pinhole (perspective) camera model 
may result in local misregistrations, which are visible as a loss of detail 
or multiple images (ghosting) (See Szeliski: Col. 3, Lns. 36-39)." This 
clearly conveys the intention of Szeliski to use perspective images as the 
input to the panoramic system. Since the images used in the image 
panoramic mosaics are perspective images, the methods taught by Seago 
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to extract three-dimensional objects from perspective images could be 
implemented with the teachings of Szeliski to extract three-dimensional 
images from a panoramic mosaic of perspective images. 

Therefore, it would have been obvious to one of ordinary skill in the art, 
at the time the invention was made, to have used the three-dimensional 
object extraction method, as taught by Seago, to extract three- 
dimensional objects from the perspective images of the panoramic 
mosaic, as taught by Szeliski, because Seago is in the same field of 
endeavor of generating three-dimensional computer graphics using image 
capture devices; and Seago expressly suggests that this method of object 
extraction produces accurate three-dimensional objects more efficiently 
than conventional systems that try to extract the three-dimensional 
objects using analytical mathematical interpretations and orthogonal 
image analysis (See Seago: Col. 11, Lns. 49-55). 

Regarding independent claim 22, Szeliski teaches a method for projecting 
texture information onto a geometric feature within an image panorama 
(See Szeliski; Fig. 2B, Item 270; Col. 27, Lns. 62-66), the method 
comprising: 
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• Receiving instructions from a user identifying a three-dimensional 
geometric surface within an image panorama (See Szeliski: Col. 27-28, 
Lns. 62-2, respectively, wherein, "The shape of the model and the 
embedding of each face into texture space are left up to the user. This 
choice can range from something as simple as a cube with six 
separate texture maps, to something as complicated as a subdivided 
dodecahedron, or even a latitude-longitude tessellated globe." Once 
the shape is determined, traditional texture-mapping is used to map 
the textures of the panoramic mosaic to the model.), the image 
panorama containing features having one or more textures (See 
Szeliski: Col. 5, Lns. 27-35); 

• Determining a directional vector from the three-dimensional geometric 
surface (See Szeliski: Fig. 15; Col. 22, Lns. 4-31); 

• Creating a geometric model of the image panorama based at least in 
part on the three-dimensional geometric surface and the directional 
vector (See Szeliski: Col. 7, Lns. 29-33 and Col. 22, Lns. 4-31. In 
column 7, Szeliski teaches the geometry of the three-dimensional 
model is a polyhedron, the faces of which constitute the three- 
dimensional geometric surfaces. In column 22, Szeliski teaches the 
direction vectors, which are used to align the images of the panoramic 
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image. Thus, the geometric model of the visual scene is dependent on 
the geometric shapes of the surfaces of the polyhedron to be texture 
mapped, and the texture map applied to the surfaces of the 
polyhedron, which is the panoramic mosaic determined using the 
direction vectors.); 

• Applying the one or more textures to the selected feature in the image 
panorama based on the geometric model (See Szeliski: Fig. 2B, Item 
270; Col. 28, Lns. 13-15, wherein Szeliski teaches, "...efficiently 
computing texture map color values for any geometry and choice of 
texture map coordinates."). 
Szeliski does not expressly disclose associating geometry information 
with a selected feature, or that the geometry information comprising 3-D 
coordinates describing the position and orientation of a selected feature 
in a reference coordinate system. Seago is cited for selecting an object 
contained within the two-dimensional perspective image for conversion 
into a three-dimensional object by a user (See Seago: Fig. 2, Item 42; Col. 
4, Lns. 46-53), as associating with it shapes or polygons that define the 
selected object's sides relates to the geometry information (See Seago: 
Fig. 2, Item 50; Col. 5, Lns. 35-38). The geometry of the 3D object is then 
oriented within the selected object's three-dimensional coordinate space 
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(See Seago: Fig. 2, Item 54; Col. 5, Lns. 50-53). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used the three-dimensional object extraction method, as 
taught by Seago, to extract three-dimensional objects from the 
perspective images of the panoramic mosaic, as taught by Szeliski, for 
the reasons provided in claim 11. 

Regarding independent claim 36, Szeliski teaches a system for 
interactively editing objects in a panoramic image, the system 
comprising: 

• A receiver for receiving one or more image panoramas representing a 
visual scene having one or more objects (See Szeliski: Fig. 2B, Item 
210; Col. 4, Lns. 66-67, wherein the invention creates full view 
panoramic mosaics from image sequences. Col. 11, Lns. 16-21 
teaches the presence of objects within the panoramic images.) and a 
point source (See Szeliski: Fig. 3, Item 310; Col. 9, Lns. 56-57, 
wherein the point source is camera 310 having its optical center fixed 
at point C); 

• A modeling module for creating a three dimensional model of the 
visual scene (See Szeliski: Col. 3, Lns. 58-61, wherein the post- 
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processing stage projects mosaics onto a convenient viewing surface, 
i.e., to create an environment map represented as a texture-mapped 
polyhedron surrounding the origin, which corresponds to a three- 
dimensional model of the visual scene.). 
Szeliski does not expressly disclose 

• Identifying a selected object in the one or more image panoramas and 
associating geometry information with the selected object, the 
geometry information comprising 3-D coordinates describing the 
position and orientation of the selected object in a reference 
coordinate system, 

• One or more interactive editing tools for providing an edit to the 
selected object; 

• A transformation module for transforming the edit relative to a 
viewpoint defined by the point source; and 

• A rendering module for projecting the transformed edit onto the 
selected object. 

Nevertheless, Seago is cited for selecting an object contained within the 
two-dimensional perspective image for conversion into a three- 
dimensional object by a user (See Seago: Fig. 2, Item 42; Col. 4, Lns. 46- 
53), as associating with it shapes or polygons that define the selected 
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object's sides relates to the geometry information (See Seago: Fig. 2, Item 
50; Col. 5, Lns. 35-38). The geometry of the 3D object is then oriented 
within the selected object's three-dimensional coordinate space (See 
Seago: Fig. 2, Item 54; Col. 5, Lns. 50-53). 

Seago also teaches generating a three-dimensional object based on the 
designated shapes performed using the graphical interface tools (See 
Seago: Figs. 9, 10, and 12; Col. 3, Lns. 61-64) and allowing a user to 
manually correct objects by using the user interface (See Seago: Col. 7, 
Lns. 56-60). 

Furthermore, Seago teaches a transforming module and a rendering 
module as software subroutines that edit an object, transform the edited 
object into 2D coordinates, and render the edited object by projecting the 
transformed object to image plane along the line-of-sight. The line-of- 
sight corresponds to the viewpoint, and therefore the transform of the 
edit occurs relative to the viewpoint defined by the point source (See 
Seago: Fig. 3; Col. 6, Lns. 1-34). 

As previously noted, the exemplary embodiment of Seago only uses a 
single input image as input for the extraction process, but does suggest 
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using two or more input images (See Seago: Col. 11, Lns. 45-47). 
Therefore, examiner believes the process of Seago would reasonable 
extract three-dimensional images from the panoramic mosaics created by 
Szeliski. 

Therefore, it would have been obvious to one of ordinary skill in the art, 
at the time the invention was made, to have used the three-dimensional 
object extraction method, as taught by Seago, to extract three- 
dimensional objects from the perspective images of the panoramic 
mosaic, as taught by Szeliski, for the reasons given in the analysis of 
claim 11. 

Regarding claim 12, as it depends from claim 11, Szeliski teaches the 
three-dimensional model comprises at least one of depth information and 
geometry information (See Szeliski: Figs. 2B, 27, 30 and 32; Col. 29, Lns. 
58-61, wherein a "3D model" implies geometric shape, as well as height, 
width, and depth.). 

Regarding claim 23, as it depends from claim 22, Szeliski teaches a 
method wherein the instructions are received using an interactive 
drawing tool (See Szeliski: Fig. 2A, Item 42; Col. 8, Lns. 30-32). 
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Regarding claim 24, as it depends from claim 22, Szeliski teaches a 
method wherein the three-dimensional geometric surface is one of a 
floor, a wall, or a ceiling (See Szeliski: Col. 27, Lns. 64-67, wherein the 
model can be a cube with six separate texture maps for each surface. 
Using an appropriate environment map would cause the top surface to 
be a ceiling, the bottom surface to be a floor, etc.). 

Regarding claim 25, as it depends from claim 22, Szeliski teaches a 
method wherein the directional vector is orthogonal to the planar surface 
(See Szeliski: Fig. 15; Col. 22, Lns. 4-6). 

Regarding claim 26, as it depends from claim 22, Szeliski teaches a 
method wherein the geometric model comprises depth information (See 
Szeliski: Figs. 27 and 30; Col. 28, Lns. 29-33). 

Regarding claim 27, as it depends from claim 22, Szeliski teaches a 
method wherein the texture information comprises color information (See 
Szeliski: Col. 28, Lns. 13-18). 

Regarding claim 28, as it depends from claim 22, Szeliski teaches a 
method comprising environment based texture mapping comprises 
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luminance information (See Szeliski: Fig. 2B; Col. 5, Lns. 27-30, wherein 
luminance is inherent to environment maps and texture maps.). 

C. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Szeliski, in view of Luken and Seago, and in further view of Blank (US Patent 
Number 5,469,536). 

Regarding claim 7, as it depends from claim 6, the combination of 
Szeliski, Luken, and Seago does not expressly suggest using edge 
detection to automatically identify the two elements. Nevertheless, Blank 
teaches detecting the edges of an object and separates portions of the 
image that are outside the edge of the object (i.e., the background 
component) from portions of the image that are inside the edge (See 
Blank: Col. 4, Lns. 17-21). The two elements are therefore identified as 
those elements within the edge, and those outside the edge. It would 
have been obvious to one of ordinary skill in the art, at the time the 
invention was made, to have used the edge detection methods, as taught 
by Blank, as an alternative to patch-based division, as taught by Szeliski, 
as modified by Luken and Seago, because it is an effective way to divide 
the image into smaller portions to conquer aligning all aspects of an 
image. 
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D. Claims 13-21 and 37 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Szeliski, in view of Seago, and in further view of Blank. 

Regarding claim 13, as it depends from claim 11, the combination of 
Szeliski and Seago does not expressly suggest receiving an edit to color 
information associated with the objects of the image. Blank is cited for 
teaching editing an image having a plurality of pixels, each pixel having a 
color, comprising the steps of selecting a plurality of colors from among 
those present in the image to form a set of selected colors, and 
manipulating only those pixels having a color defined in the set of 
selected colors so as to change a visual feature of the image (See Blank: 
Col. 6, Lns. 29-47). It would have been obvious to one of ordinary skill in 
the art, at the time the invention was made, to have allowed modification 
of the colors within an image, as taught by Blank, to manipulate the 
colors within the images used to create the panoramic image mosaics, as 
taught by Szeliski, as modified by Seago, because it would enable the 
user to quickly and efficiently modify or enhance the appearance of an 
image to desired goal (See Blank: Col. 6, Lns. 23-28). 

Regarding claim 14, as it depends from claim 11, the combination of 
Szeliski and Seago does not expressly suggest receiving an edit to alpha 
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information associated with the objects of the image. Blank is cited for 
teaching determining whether to reset the transparency alpha-bit flag. If 
the transparency flag is reset, it is set to opaque (See Blank: Fig. 11; Col. 
20, Lns. 61-66). It would have been obvious to one of ordinary skill in the 
art, at the time the invention was made, to have allowed modification of 
the alpha values within an image, as taught by Blank, to control the 
presentation of the images of the panoramic image mosaics, as taught by 
Szeliski, as modified by Seago, for the same reasons given in claim 13. 

Regarding claim 15, as it depends from claim 11, the combination of 
Szeliski and Seago does not expressly suggest receiving an edit to depth 
information associated with the objects of the image. Blank is cited for 
moving several objects to various Z depth layers (See Blank: Col. 13, Lns. 
8-16). It would have been obvious to one of ordinary skill in the art, at 
the time the invention was made, to have moved objects in three- 
dimensions, as taught by Blank, to manipulate the objects extracted 
from the mosaics, as taught by the combination of Szeliski and Seago, for 
the same reasons given in claim 13. 

Regarding claim 16, as it depends from claim 11, the combination of 
Szeliski and Seago does not expressly suggest receiving an edit to 
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geometry information associated with the objects of the image. Blank is 
cited for teaching trimming, which allows the user to trim off any 
undesired edges of an object to reveal the background below (See Blank: 
Col. 47, Lns. 11-20). It would have been obvious to one of ordinary skill 
in the art, at the time the invention was made, to have used a 
manipulation tools, as taught by Blank, to manipulate the objects 
extracted from the panoramic image mosaics, as taught by Szeliski, as 
modified by Seago, for the same reasons given in claim 13. 

Regarding claim 17, as it depends from claim 11, the combination of 
Szeliski and Seago does not expressly suggest providing a user with an 
interactive drawing tool that specifies edits for one or more objects of the 
image or receiving the edits made by the user using the interactive 
drawing tool. Blank is cited for teaching function calls, wherein 
attributes to be changed are pre-selected or selected by the user. When 
function is accessed by a user call, the system operates as a highly 
interactive and very powerful image editing tool. The function accesses 
preset gamma attributes, layer, and object or area selections, e.g., all 
pixels on layer one with a hue of blue at a value range of 10 to 75 (See 
Blank: Col. 21, Lns. 18-23) It would have been obvious to one of ordinary 
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skill in the art, at the time the invention was made, to have used 
manipulation tools, as taught by Blank, to manipulate the objects 
extracted from the panoramic image mosaics, as modified by Seago, for 
the same reasons given in claim 13. 

Regarding claims 18 and 37, as they depend from claims 17 and 36, 
respectively, the combination of Szeliski and Seago does not expressly 
suggest the interactive drawing tool is one of an extrusion tool, a ground 
plane tool, a depth chisel tool, and a non-uniform rational B-spline tool. 
Blank is cited for teaching trimming, which allows the user to trim off 
any undesired edges of an object to reveal the background below, which 
corresponds to the depth chisel tool (See Blank: Col. 47, Lns. 11-20). It 
would have been obvious to one of ordinary skill in the art, at the time 
the invention was made, to have used a manipulation tools, as taught by 
Blank, to manipulate the objects extracted from the panoramic image 
mosaics, as taught by Szeliski, as modified by Seago, for the same 
reasons given in claim 13. 

Regarding claim 19, as it depends from claim 17, the combination of 
Szeliski and Seago does not expressly suggest interactive drawing tool 
specifies a selected value for depth for objects of the image. Blank is cited 
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for teaching tests to see if an object is at the top layer (Z=31) or at the 
bottom layer (Z=0) (See Blank: Col. 22, Lns. 35-62). It would have been 
obvious to one of ordinary skill in the art, at the time the invention was 
made, to have used a manipulation tools, as taught by Blank, to 
manipulate the objects extracted from the panoramic image mosaics, as 
taught by Szeliski, as modified by Seago, for the same reasons given in 
claim 13. 

Regarding claim 20, as it depends from claim 17, the combination of 
Szeliski and Seago does not expressly suggest the interactive drawing 
tool incrementally adds to the depth for objects of the image. Blank is 
cited for teaching area addition, which allows the user to add additional 
selections to the current selection. Once the user clicks the area addition 
tool, "add" mode is retained until it is clicked again or switched to a 
different selection tool (drag box, drag free, color select, or brush select) 
(See Blank: Col. 34, Lns. 8-11). It would have been obvious to one of 
ordinary skill in the art, at the time the invention was made, to have 
used a manipulation tools, as taught by Blank, to manipulate the objects 
extracted from the panoramic image mosaics, as taught by Szeliski, as 
modified by Seago, for the same reasons given in claim 13. 
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Regarding claim 21, as it depends from claim 17, the combination of 
Szeliski and Seago does not expressly suggest the interactive drawing 
tool incrementally subtracts from the depth for objects of the image. 
Blank is cited for teaching area subtraction, which allows the user to 
deselect certain regions of the currently selected area. Click on the Area 
Subtraction tool and select areas to be subtracted. Again, "subtract" 
mode is retained until it is clicked again or switched to a different 
selection tool (drag box, drag free, color select, or brush select) (See 
Blank: Col. 34, Lns. 22-26). It would have been obvious to one of 
ordinary skill in the art, at the time the invention was made, to have 
used a manipulation tools, as taught by Blank, to manipulate the objects 
extracted from the panoramic image mosaics, as taught by Szeliski, as 
modified by Seago, for the same reasons given in claim 13. 

Conclusion 

Any inquiry concerning this communication or earlier communications 
from the examiner should be directed to Crystal Murdoch whose 
telephone number is (571) 270-1043. The examiner can normally be 
reached on Mon. - Fri. 10:00am - 6:30pm. If attempts to reach the 
examiner by telephone are unsuccessful, the examiner's supervisor, 
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Mark Zimmerman can be reached on (571) 272-7653. The fax phone 
number for the organization where this application or proceeding is 
assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status 
information for published applications may be obtained from either 
Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more 
information about the PAIR system, see http://pair-direct.uspto.gov. 
Should you have questions on access to the Private PAIR system, contact 
the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you 
would like assistance from a USPTO Customer Service Representative or 
access to the automated information system, call 800-786-9199 (IN USA 
OR CANADA) or 571-272-1000. 

/CM/ Crystal Murdoch 

Examiner 
Art Unit 2628 



/ Mark Zimmerman / 
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Supervisory Patent Examiner 
Tech Center 2600 



